Papers with multimodal information integration
Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. |
| Approach: | They propose a task that focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. |
| Outcome: | The proposed framework improves transcript quality through post-editing and improves performance over speech-only baselines. |